Course content data analysis and prediction

ABSTRACT

Systems and methods of the present invention provide for receiving, from a content activity agent on a first client device, a content usage data; input, into a machine learning model, at least one content activity parameter, translated from the content usage data; generating, using an output from the machine learning model, a graphical user interface (GUI), displayed on a second client device, and including a report of: the output, a recommendation for an update to a course content, and a prediction of an increase to an average assessment score associated with the course content if the updated content is used.

FIELD OF THE INVENTION

This disclosure relates to the field of systems and methods configuredto receive a plurality of user behavior data relating to a structure ofa course content, analyze the data to determine assessment score dataand user interaction time data, input the user behavior data into amachine learning model, and generate reports, predictions, and/orrecommendations for improving user performance for the course content.

SUMMARY OF THE INVENTION

The present invention provides systems and methods comprising one ormore server hardware computing devices or client hardware computingdevices, communicatively coupled to a network, and each comprising atleast one processor executing specific computer-executable instructionswithin a memory that, when executed, cause the system to: receive, froma content activity agent on a first client device, a content usage data;input, into a machine learning model, at least one content activityparameter, translated from the content usage data; generate, using anoutput from the machine learning model, a graphical user interface(GUI), displayed on a second client device, and including a report of:the output, a recommendation for an update to a course content, and aprediction of an increase to an average assessment score associated withthe course content if the updated content is used.

The above features and advantages of the present invention will bebetter understood from the following detailed description taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system level block diagram for analyzing coursecontent data, and generating reports, predictions, and/orrecommendations for a course content.

FIG. 2 illustrates a system level block diagram for analyzing coursecontent data, and generating reports, predictions, and/orrecommendations for a course content.

FIG. 3 illustrates a system level block diagram for analyzing coursecontent data, and generating reports, predictions, and/orrecommendations for a course content.

FIG. 4 illustrates a flow diagram including process steps for capturingand storing user assessment data associated with a course content data.

FIG. 5 illustrates a flow diagram including process steps for capturingand storing user usage time data associated with a course content data.

FIG. 6 illustrates a flow diagram including process steps forgenerating, from a machine learning output, reports, predictions, and/orrecommendations for improving user performance for the course content.

FIG. 7 illustrates a non-limiting example user interface for displayinguser assessment data associated with a course content data.

FIG. 8 illustrates a non-limiting example user interface for displayinguser usage time data associated with a course content data.

FIG. 9 illustrates a non-limiting example user interface for displayingreports, predictions, and/or recommendations for improving userperformance for the course content.

DETAILED DESCRIPTION

The present inventions will now be discussed in detail with regard tothe attached drawing figures that were briefly described above. In thefollowing description, numerous specific details are set forthillustrating the Applicant's best mode for practicing the invention andenabling one of ordinary skill in the art to make and use the invention.It will be obvious, however, to one skilled in the art that the presentinvention may be practiced without many of these specific details. Inother instances, well-known machines, structures, and method steps havenot been described in particular detail in order to avoid unnecessarilyobscuring the present invention. Unless otherwise indicated, like partsand method steps are referred to with like reference numerals.

When learners (e.g., college students) attends a learning course, it maybe assumed that their ultimate goal is to achieve the highest possiblegrade for the learning course. Similarly, a course content creator(e.g., a professor at a university) also presumably wants the highestpossible grades for their learners.

Many institutions or entities, such as universities, provide coursecontent creators with course creation tools for learning courses. (e.g.,Pearson Education's MEMPHIS). These course creation tools often allow acourse content creator to access a software running on a client deviceto select from one or more content creation structures (e.g., contenttemplates, components, modules, formats, layouts, etc.), which learningstakeholders, such as universities, professors, administrators, coursecontent creators, and the like, use to generate the course content.

However, current course content creation tools only allow for thecreation of course content (e.g., by creating course content fromscratch, cloning the content for an existing “master” course, generatinga class from existing content creation structures, etc.). After learnersaccess the content created by such course content creation tools,current course creation tools are unable to recognize user behavioralpatterns from the selected content structure within specific learningcourses, and therefore do not provide the course content creators, orfuture course content creators, with the results of the learners'performance in the use of such content structures. They are thereforeunable to predict the success of any given content structure when usedin the future and have no guidance regarding which content structuresare the most effective.

Current content creation tools are therefore unable to determine andrecommend future course content. Furthermore, even if such informationwas available via current course content creation tools, contentcreators for the learning courses would have no way of accessing,viewing, or otherwise determining how learners performed using aparticular content structure, in order to determine an optimal templateor template component to use in future learning courses. In short, whena course content creator creates and delivers course content to bepresented to learners, they lack sufficient content usage data andresources to determine which course content structure should be includedwithin an existing learning course, or in future courses, or torecommend the content for future courses in order to optimize learnerperformance within the learning course in order to create the highestpossible results.

To overcome the discrepancies in the prior art, the disclosedembodiments observe student behaviors when interacting with elements andstructures of learning course content for a learning course, thencollect and analyze the data associated with such user interactions andbehaviors. This analyzed data is then converted into parameters to beinput into a machine learning model in order to execute a machinelearning algorithm, which processes the information in light of trainingdata provided, and outputs data allowing for better content designpractices, resulting in a better customer (e.g., content creator,instructor, and/or student) experience.

This output may then be used by content authors to: 1. Predict anddesign courses that can result in higher student performance in thecourse (e.g., higher “Check Your Understanding, or CYU, results) andultimately higher final grades for the course; 2. View detailed data ofthe exiting course usage data; 3. Predict and recommend future contenttemplates, which may not currently be available, and should be added toavailable templates for learning courses, which will result in higherperformance results for the learners; and 4. Provide recommendations tocontent authors for a better content design experience that will resultin comparatively higher grades.

FIG. 1 illustrates a non-limiting example distributed computingenvironment 100, which includes one or more computer server computingdevices 102, one or more client computing devices 106, and othercomponents that may implement certain embodiments and features describedherein. Other devices, such as specialized sensor devices, etc., mayinteract with client 106 and/or server 102. The server 102, client 106,or any other devices may be configured to implement a client-servermodel or any other distributed computing architecture.

Server 102, client 106, and any other disclosed devices may becommunicatively coupled via one or more communication networks 120.

Communication network 120 may be any type of network known in the artsupporting data communications. As non-limiting examples, network 120may be a local area network (LAN; e.g., Ethernet, Token-Ring, etc.), awide-area network (e.g., the Internet), an infrared or wireless network,a public switched telephone networks (PSTNs), a virtual network, etc.Network 120 may use any available protocols, such as (e.g., transmissioncontrol protocol/Internet protocol (TCP/IP), systems networkarchitecture (SNA), Internet packet exchange (IPX), Secure Sockets Layer(SSL), Transport Layer Security (TLS), Hypertext Transfer Protocol(HTTP), Secure Hypertext Transfer Protocol (HTTPS), Institute ofElectrical and Electronics (IEEE) 802.11 protocol suite or otherwireless protocols, and the like.

The embodiments shown in FIGS. 1-2 are thus one example of a distributedcomputing system and is not intended to be limiting. The subsystems andcomponents within the server 102 and client devices 106 may beimplemented in hardware, firmware, software, or combinations thereof.Various different subsystems and/or components 104 may be implemented onserver 102. Users operating the client devices 106 may initiate one ormore client applications to use services provided by these subsystemsand components. Various different system configurations are possible indifferent distributed computing systems 100 and content distributionnetworks. Server 102 may be configured to run one or more serversoftware applications or services, for example, web-based or cloud-basedservices, to support content distribution and interaction with clientdevices 106. Users operating client devices 106 may in turn utilize oneor more client applications (e.g., virtual client applications) tointeract with server 102 to utilize the services provided by thesecomponents. Client devices 106 may be configured to receive and executeclient applications over one or more networks 120. Such clientapplications may be web browser-based applications and/or standalonesoftware applications, such as mobile device applications. Clientdevices 106 may receive client applications from server 102 or fromother application providers (e.g., public or private applicationstores).

As shown in FIG. 1, various security and integration components 108 maybe used to manage communications over network 120 (e.g., a file-basedintegration scheme or a service-based integration scheme). Security andintegration components 108 may implement various security features fordata transmission and storage, such as authenticating users orrestricting access to unknown or unauthorized users,

As non-limiting examples, these security components 108 may comprisededicated hardware, specialized networking components, and/or software(e.g., web servers, authentication servers, firewalls, routers,gateways, load balancers, etc.) within one or more data centers in oneor more physical location and/or operated by one or more entities,and/or may be operated within a cloud infrastructure.

In various implementations, security and integration components 108 maytransmit data between the various devices in the content distributionnetwork 100. Security and integration components 108 also may use securedata transmission protocols and/or encryption (e.g., File TransferProtocol (FTP), Secure File Transfer Protocol (SFTP), and/or Pretty GoodPrivacy (PGP) encryption) for data transfers, etc.).

In some embodiments, the security and integration components 108 mayimplement one or more web services (e.g., cross-domain and/orcross-platform web services) within the content distribution network100, and may be developed for enterprise use in accordance with variousweb service standards (e.g., the Web Service Interoperability (WS-I)guidelines). For example, some web services may provide secureconnections, authentication, and/or confidentiality throughout thenetwork using technologies such as SSL, TLS, HTTP, HTTPS, WS-Securitystandard (providing secure SOAP messages using XML encryption), etc. Inother examples, the security and integration components 108 may includespecialized hardware, network appliances, and the like (e.g.,hardware-accelerated SSL and HTTPS), possibly installed and configuredbetween servers 102 and other network components, for providing secureweb services, thereby allowing any external devices to communicatedirectly with the specialized hardware, network appliances, etc.

Computing environment 100 also may include one or more data stores 110,possibly including and/or residing on one or more back-end servers 112,operating in one or more data centers in one or more physical locations,and communicating with one or more other devices within one or morenetworks 120. In some cases, one or more data stores 110 may reside on anon-transitory storage medium within the server 102. In certainembodiments, data stores 110 and back-end servers 112 may reside in astorage-area network (SAN). Access to the data stores may be limited ordenied based on the processes, user credentials, and/or devicesattempting to interact with the data store.

With reference now to FIG. 2, a block diagram of an illustrativecomputer system is shown. The system 200 may correspond to any of thecomputing devices or servers of the network 100, or any other computingdevices described herein. In this example, computer system 200 includesprocessing units 204 that communicate with a number of peripheralsubsystems via a bus subsystem 202. These peripheral subsystems include,for example, a storage subsystem 210, an I/O subsystem 226, and acommunications subsystem 232.

One or more processing units 204 may be implemented as one or moreintegrated circuits (e.g., a conventional micro-processor ormicrocontroller), and controls the operation of computer system 200.These processors may include single core and/or multicore (e.g., quadcore, hexa-core, octo-core, ten-core, etc.) processors and processorcaches. These processors 204 may execute a variety of resident softwareprocesses embodied in program code and may maintain multipleconcurrently executing programs or processes. Processor(s) 204 may alsoinclude one or more specialized processors, (e.g., digital signalprocessors (DSPs), outboard, graphics application-specific, and/or otherprocessors).

Bus subsystem 202 provides a mechanism for intended communicationbetween the various components and subsystems of computer system 200.Although bus subsystem 202 is shown schematically as a single bus,alternative embodiments of the bus subsystem may utilize multiple buses.Bus subsystem 202 may include a memory bus, memory controller,peripheral bus, and/or local bus using any of a variety of busarchitectures (e.g. Industry Standard Architecture (ISA), Micro ChannelArchitecture (MCA), Enhanced ISA (EISA), Video Electronics StandardsAssociation (VESA), and/or Peripheral Component Interconnect (PCI) bus,possibly implemented as a Mezzanine bus manufactured to the IEEE P1386.1standard).

I/O subsystem 226 may include device controllers 228 for one or moreuser interface input devices and/or user interface output devices,possibly integrated with the computer system 200 (e.g., integratedaudio/video systems, and/or touchscreen displays), or may be separateperipheral devices which are attachable/detachable from the computersystem 200. Input may include keyboard or mouse input, audio input(e.g., spoken commands), motion sensing, gesture recognition (e.g., eyegestures), etc.

As non-limiting examples, input devices may include a keyboard, pointingdevices (e.g., mouse, trackball, and associated input), touchpads, touchscreens, scroll wheels, click wheels, dials, buttons, switches, keypad,audio input devices, voice command recognition systems, microphones,three dimensional (3D) mice, joysticks, pointing sticks, gamepads,graphic tablets, speakers, digital cameras, digital camcorders, portablemedia players, webcams, image scanners, fingerprint scanners, barcodereaders, 3D scanners, 3D printers, laser rangefinders, eye gaze trackingdevices, medical imaging input devices, MIDI keyboards, digital musicalinstruments, and the like.

In general, use of the term “output device” is intended to include allpossible types of devices and mechanisms for outputting information fromcomputer system 200 to a user or other computer. For example, outputdevices may include one or more display subsystems and/or displaydevices that visually convey text, graphics and audio/video information(e.g., cathode ray tube (CRT) displays, flat-panel devices, liquidcrystal display (LCD) or plasma display devices, projection devices,touch screens, etc.), and/or non-visual displays such as audio outputdevices, etc. As non-limiting examples, output devices may include,indicator lights, monitors, printers, speakers, headphones, automotivenavigation systems, plotters, voice output devices, modems, etc.

Computer system 200 may comprise one or more storage subsystems 210,comprising hardware and software components used for storing data andprogram instructions, such as system memory 218 and computer-readablestorage media 216.

System memory 218 and/or computer-readable storage media 216 may storeprogram instructions that are loadable and executable on processor(s)204. For example, system memory 218 may load and execute an operatingsystem 224, program data 222, server applications, client applications220, Internet browsers, mid-tier applications, etc.

System memory 218 may further store data generated during execution ofthese instructions. System memory 218 may be stored in volatile memory(e.g., random access memory (RAM) 212, including static random accessmemory (SRAM) or dynamic random access memory (DRAM)). RAM 212 maycontain data and/or program modules that are immediately accessible toand/or operated and executed by processing units 204.

System memory 218 may also be stored in non-volatile storage drives 214(e.g., read-only memory (ROM), flash memory, etc.) For example, a basicinput/output system (BIOS), containing the basic routines that help totransfer information between elements within computer system 200 (e.g.,during start-up) may typically be stored in the non-volatile storagedrives 214.

Storage subsystem 210 also may include one or more tangiblecomputer-readable storage media 216 for storing the basic programmingand data constructs that provide the functionality of some embodiments.For example, storage subsystem 210 may include software, programs, codemodules, instructions, etc., that may be executed by a processor 204, inorder to provide the functionality described herein. Data generated fromthe executed software, programs, code, modules, or instructions may bestored within a data storage repository within storage subsystem 210.

Storage subsystem 210 may also include a computer-readable storage mediareader connected to computer-readable storage media 216.Computer-readable storage media 216 may contain program code, orportions of program code. Together and, optionally, in combination withsystem memory 218, computer-readable storage media 216 maycomprehensively represent remote, local, fixed, and/or removable storagedevices plus storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation.

Computer-readable storage media 216 may include any appropriate mediaknown or used in the art, including storage media and communicationmedia, such as but not limited to, volatile and non-volatile, removableand non-removable media implemented in any method or technology forstorage and/or transmission of information. This can include tangiblecomputer-readable storage media such as RAM, ROM, electronicallyerasable programmable ROM (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disk (DVD), or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or other tangible computer readablemedia. This can also include nontangible computer-readable media, suchas data signals, data transmissions, or any other medium which can beused to transmit the desired information, and which can be accessed bycomputer system 200.

By way of example, computer-readable storage media 216 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 216 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 216 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magneto-resistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 200.

Communications subsystem 232 may provide a communication interface fromcomputer system 200 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 2, thecommunications subsystem 232 may include, for example, one or morenetwork interface controllers (NICs) 234, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 236, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally, and/or alternatively, the communicationssubsystem 232 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, Fire Wire® interfaces, USB® interfaces, and the like.Communications subsystem 236 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

In some embodiments, communications subsystem 232 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 200. For example,communications subsystem 232 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators). Additionally, communications subsystem 232 maybe configured to receive data in the form of continuous data streams,which may include event streams of real-time events and/or event updates(e.g., sensor data applications, financial tickers, network performancemeasuring tools, clickstream analysis tools, automobile trafficmonitoring, etc.). Communications subsystem 232 may output suchstructured and/or unstructured data feeds, event streams, event updates,and the like to one or more data stores that may be in communicationwith one or more streaming data source computers coupled to computersystem 200.

The various physical components of the communications subsystem 232 maybe detachable components coupled to the computer system 200 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 200. Communicationssubsystem 232 also may be implemented in whole or in part by software.

Due to the ever-changing nature of computers and networks, thedescription of computer system 200 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

As noted above, the disclosed embodiments observe student behaviors wheninteracting with elements and structures of learning course content fora learning course, then collect and analyze the data associated withsuch user interactions and behaviors. This analyzed data is thenconverted into parameters for a machine learning model/algorithm, whichprocesses the information in light of training data provided, andoutputs data allowing for better content design practices, resulting ina better customer (e.g., content creator, instructor, and/or learner)experiences.

In the disclosed embodiments course content creators, may be providedaccess to improved and more efficient course creation tools via alearning management system (LMS), however, although course creationtools may be available within the LMS to a certain extent, not allcourse creation tools disclosed herein are found in the LMS. As anon-limiting example, external authoring tools may also be utilized tocreate course content, and these or the content generated from them maybe made available through the LMS. In some disclosed embodiments thedisclosed LMS may allow a content creator to select a most effectivecourse content structure (e.g., content templates, components, modules,formats, layouts, rubrics, blueprints, media types associated with theeducational content, etc.) that provide students with the highestpotential grades.

In the disclosed embodiments, learning institutions (e.g., universities)may provide instructors or other course content creators with an LMSthat includes one or more authoring or other support tools in order tocreate course content. Additionally, LMS can support course authoringfor a certain level. LMS can show course content authored by otherauthoring tools like Pearson Education's MEMPHIS. In this context, thecontent created from a course authoring tool such as MEMPHIS, are shownin the LMS. Learners may also use the LMS to access learner supporttools, which may be used to access assignments, assessments, varioustext or multimedia content, etc. in order to progress through thelearning course.

For each course, a course blueprint, including course foundationmaterial (CFM), may be provided, which may include required curriculumor the platform for learning course content that the instructors need toteach. Non-limiting examples of such CFM may include a “master” coursecontent for each course offered by the institution, which may include afoundation content for the course.

In some embodiments, the CFM may be broken down, for each course, into aseries of topics, each of which may have a sequence of LP pages,disclosed below. Each lesson may further include subjects, and/orlearning modules. Each of the topics may further include a collection oflearning presentation (LP) pages, each including one or more elementsthat make up the LP page. Each topic may therefore be made up of asequence of LP pages. As a non-limiting example, each of these LP pagesmay be an HTML page made up of text, multi-media content (e.g., imagesor video links), external links, and/or other external content.

In some embodiments, the CFM may be organized into a uniform learningflow. As a non-limiting example, in some embodiments, each topic in alearning course may be organized into a learning flow including anintroduction, one or more LP pages, one or more CYU pages or quizzes,and a summary/review for all previously presented material.

The introduction may include the introductory details for the topic. TheLP pages may include the presentation layer—the main educational contentand structure by which students view the course contents, and each ofthese pages may include content, such as text, multi-media presentation(e.g., images, video content, other media types, etc.) of theeducational content, external links, links to external content, and thelike to deliver the presentation to the learners. The CYU pages may beassessments within the learning flow that evaluate the level ofunderstanding of the content by the students. The review pages mayprovide a summary on the content delivered by the topic.

In some embodiments, the institution may create and provide, through thecourse authoring tool, templates for each topic within a learning coursethat reflects the learning flow described above. As a non-limitingexample, the institution may provide a certain number of LP templatesand a certain number of CYU templates. The course content creator maythen select from among these templates to create their course anddetermine how the course content will be presented to the learners inthe course. In some embodiments, the course content creator may selectfrom templates for the course, templates for each topic, templates foreach LP page, templates for each content element in each learning page,templates for each CYU page, and the like.

The instructor or other content creator may then access the authoringtool, or any other available course content creation tools, and designtheir course according to a content structure that they are confidentwill provide the best results.

Thus, when an instructor or other course content creator creates acourse, the course creator may choose the course content to be presentedto the learners through the learning course. This may be accomplished bythe course creator by selecting and cloning a master course through theLMS, the course authoring tool, or MEMPHIS, and creating a copy of thatmaster course. Alternatively, the course creator may select and edit themaster course CFM and LP pages based on the content that they plan topresent to the learners. Finally, if a course creator is creating thecourse from scratch, the course creator may create a blueprint of theCFM and the content for each LP page for the course, possibly, using thetemplates or other available editable content at a course, topic, page,or element level, and arranging the content structure in a customizedmanner.

Once the content of each learning course has been completed by thecourse content creator, the disclosed system may be configured tocapture course usage data, transmit the captured and collected courseusage data to server 112, which may execute an evaluation and predictionengine comprising a machine learning model/algorithm, which outputs theevaluation, predictions of the most efficient templates, and recommendedtemplate usage for the most effective results.

This may be implemented in two phases: First, the LP pages may include,or may associated with, one or more software modules in the LMS makingup one or more course content usage agents 300, described in more detailbelow, which may by execute in a browser, and may asynchronously collectusage data and/or consumption data and transmit the collected datathrough server 112, to database 110. The server 112 may then execute themachine learning algorithm 305 (e.g., a regression method) using themachine learning model 310. In some embodiments, these agents may existin the course content file server 112 and may be loaded to a browserwhen the content files are requested from the LMS browser. These agentsmay then be executed in the browser and may send consumption data to theserver 112. The second phase includes inputting the collected data,translated into parameters as needed, into the machine learning model310, and executing the machine learning algorithm 305 to generateanalysis, predictions, and recommendations for the most effectivecontent or content components, based on the collected dataset. Thedisclosed embodiments may then output a report (e.g., displayed on aGUI) including the results and tools to evaluate and predict resultsbased on the data collected, and select a most effective contentstructure (e.g., templates).

As noted above, software modules comprising course content usage agents300 may be included within, or associated with pages or other elementsof the course content, thereby overcoming the lack in the prior art ofan ability to track usage patterns in order to determine the learningeffectiveness of pages, page elements, topics, and/or courses within alearning course. These agents may be stored in association with filecontent stored within the course content file server 112, and when theLMS loads in the browser, those agents are loaded into the browser.These course content usage agents 300 may monitor user activity,generate data from user activity and assessments (e.g., CYU pages), andtransmit this data to server 112 for analysis. Specifically, server 112may analyze the data to determine the effectiveness of the contentstructure of each page, page element, topic, learning course, and so on.In some distributed server embodiments, server(s) 112 may include one ormore hosting file servers, which, in some embodiments, may be separatefrom one or more data collecting servers.

In some embodiments, and as a non-limiting example, each course contentusage agent 300 may comprise one or more JavaScript files associatedwith one or more dynamic HTML pages that make up the LP pages in thetopic and/or the learning course. These JavaScript agents may executesoftware instructions to identify, gather, track, and organize userbehavior and performance data, and transmit the aggregated data toserver 112 for analysis.

The software instructions within each course content usage agent 300 maybe configured to evaluate the course content structure in the context ofone or more user behaviors. In some embodiments, course usage data mayinclude: 1. Check Your Understanding (CYU), which may be tracked andused to help students understand their current level of understanding ofthe course usage; and 2.Time spent with user interactions with aparticular LP page (or LP page elements within that LP page), which mayindicate an amount of interaction between the learner and the coursecontent.

The course content usage agent 300 may therefore collect, track,analyze, and/or transmit to server 112, grade or score information(e.g., CYU scores associated with a specific course content page and/orelement), as an assessment of a learner's performance for atopic/module, or as an average across a learning course. Likewise, thecourse content usage agent 300 may further collect, track, analyze,and/or transmit to the server, the amount of time that a user spends onan LP page (and/or its individual components, such as text, images,links, etc.).

As the purpose of the disclosed embodiments is to determine, for aparticular course content structure and/or component, whether theprediction algorithm will identify that component as an effectiveelement for learner success, each component or element may be associatedwith an identifier (id—e.g., a course id, lesson id, template id, atemplate element id, etc.). While collecting, tracking, analyzing, andtransmitting data, the course content usage agent 300 may therefore alsoidentify associated ids for each component or element, identifiers.

For example, the course content usage agent 300 may be configured torecognize a learning course id associated with any elements within thecourse content structure, and any associated characteristics describedbelow may also be associated with the learning course id. Similarly, thecourse content usage agent 300 may be configured to recognize a lessonid for a lesson within the learning course, associated with any elementsof the lesson. The characteristics described below may thereforelikewise be associated with the lesson id. The course content usageagent 300 may also be configured to recognize a template id for atemplate within the learning course. The characteristics described belowmay likewise be associated with the template id.

Thus, the course content usage agent 300 may capture the following datain the workflow from the student LMS. For the time capturing workflow,the course content usage agent 300 may capture: 1. An LP Page id; 2 atimestamp indicating when the LP page is loaded; and 3. A timestampindicating when the LP page is unloaded. For the CYU questions workflow,the course content usage agent 300 may capture the following data: 1. AnLP Page id; and 2. The answer to a CYU question provided by the user.

Turning now to FIG. 4, a flow diagram of steps to determine CYU scoresand store the CYU score data and associated identifier data isdemonstrated. In step 400, a user may log into the LMS to access thelearning course. In step 410, the user may access the CYU questions inthe CYU page, and attempt to answer them. In step 420, the CYU gradesare sent to a data repository in association with the associatedidentifiers.

Turning now to FIG. 5, a flow diagram of steps to determine userinteraction times and store the user interaction time data andassociated identifier data is demonstrated. In step 500, a user may loginto the LMS to access the learning course. In step 510, the user mayaccess a particular LP page within the learning course. In step 520, thepage access time and page identifier data may be sent to a datarepository by the LMS browser. In step 530, the user may leave the LPpage, and in step 540, the time that the user left the LP page and thepage identifier data may be sent to the data repository by the LMSbrowser.

The course content usage agent 300 may further analyze the coursecontent layout information. As a non-limiting example, the agent mayanalyze the details available from the course authoring tool regardingspecific content page information available from pages created using theavailable templates.

The course content usage agent 300 may then identify, within itsassociated page, page element, topic, and/or course, one or moregranular elements or components that define the user behavior and/orperformance of an individual LP or CYU page. As above, the coursecontent usage agent 300 for the page for each of these elements maycollect, analyze, and/or transmit data about these elements to server112, which may convert the characteristics into features, features sets,feature vectors, and/or parameters to be input into a machine learningmodel 310 for execution of a machine learning algorithm 305 to identifythe most effective predicted course content structures.

Non-limiting examples of additional elements within the LP page (alsoreferred to as supportive supplemental link types) may include images,video, Uniform Resource Locators (URLs), simulations, text documents,and internal URLs. Characteristics for additional elements may includetext length, image area, external supplemental added factor, externalimages, external links, external videos, external URLs, and externaltext documents.

For example, the agent may identify one or more text lengthcharacteristics, which may identify an amount and/or length of blocks oftext (i.e., the size of the textual content) within each of one or moreindividual elements of an LP or content for a learning course. The agentmay further identify an associated course, lesson, template, learningpage, structure component, etc. In some embodiments, the agent mayfurther identify an amount of time that a user spends interacting withthe element(s), as described above. The agent may then transmit the textlength characteristic(s), associated identifiers, and/or interactiontimes to the server 112.

The agent may also identify one or more image area characteristics,which may identify an image area for images (i.e., the width and theheight of the area of the images, possibly in pixels, defining the pixelspace used by the images) within each of one or more individual elementsof an LP page or other content for a learning course. If there aremultiple images, the image area data for all or a combination of theimages within a particular page may be used to determine the image area.The agent may further identify an associated course, lesson, template,learning page, structure component, etc. In some embodiments, the agentmay further identify an amount of time that a user spends interactingwith the element(s). The agent may then transmit the image areacharacteristic(s), associated identifiers, and/or interaction times tothe server 112.

The agent may also identify one or more external supplemented addedfactors characteristics, which may identify external added factors(e.g., external course materials) within each of one or more individualelements of an LP page or content for a learning course. The agent mayfurther identify an associated course, lesson, template, learning page,structure component, etc. In some embodiments, the agent may furtheridentify an amount of time that a user spends interacting with theelement(s). The agent may then transmit the external supplemented addedfactor characteristic(s), associated identifiers, and/or interactiontimes to the server 112.

The agent may also identify one or more external image characteristics,which may identify external images, or a number of external imageswithin each of one or more individual elements of an LP page or contentfor a learning course. The agent may further identify an associatedcourse, lesson, template, learning page, structure component, etc. Insome embodiments, the agent may further identify an amount of time thata user spends interacting with the element(s). The agent may thentransmit the external image characteristic(s), associated identifiers,and/or interaction times to the server 112.

The agent may also identify one or more external link characteristics,which may identify one or more external links and/or a number ofexternal links that a user interacts with (e.g., use or views), withineach of one or more individual elements of an LP page or content for alearning course. The agent may further identify an associated course,lesson, template, learning page, structure component, etc. In someembodiments, the agent may further identify an amount of time that auser spends interacting with the element(s). The agent may then transmitthe external link characteristic(s), associated identifiers, and/orinteraction times to the server 112.

The agent may also identify one or more external video characteristics,which may identify one or more external videos and/or a number ofexternal videos that a user interacts with (e.g., use or views), withineach of one or more individual elements of an LP page or content for alearning course. The agent may further identify an associated course,lesson, template, learning page, structure component, etc. In someembodiments, the agent may further identify an amount of time that auser spends interacting with the element(s). The agent may then transmitthe external video characteristic(s), associated identifiers, and/orinteraction times to the server 112.

The agent may also identify one or more Uniform Resource Locator (URL)characteristics, which may identify one or more external URLs and/or anumber of external URLs that a user interacts with, within each of oneor more individual elements of an LP page or content for a learningcourse. The agent may further identify an associated course, lesson,template, learning page, structure component, etc. In some embodiments,the agent may further identify an amount of time that a user spendsinteracting with the element(s). The agent may then transmit theexternal URL characteristic(s), associated identifiers, and/orinteraction times to the server 112.

The agent may also identify one or more external textual documentcharacteristics, which may identify one or more external textualdocuments and/or a number of external textual documents that a userinteracts with (e.g., use or views), within each of one or moreindividual elements of an LP or content for a learning course. The agentmay further identify an associated course, lesson, template, learningpage, structure component, etc. In some embodiments, the agent mayfurther identify an amount of time that a user spends interacting withthe element(s). The agent may then transmit the external textualdocument characteristic(s), associated identifiers, and/or interactiontimes to the server 112.

Once the course content usage agent 300 has aggregated all information(e.g., CYU grades, interaction time, data from individual LP pages orassociated elements, etc.), the agent may be configured to transmit theaggregated data to the server 112 for analysis.

Server 112 may receive the data from one or more course content usageagents 300 and perform analysis of the data with the support ofpredictive analysis associated with components associated with the data.As a non-limiting example, the analysis of the data received from theagents may represent a type of market research that defines the contentthat is most consumed and most successful (i.e., results in the highestCYU scores and usage).

To make a determination from this analysis, server may identify,analyze, and configure the data to be input as parameters into a machinelearning model 310, in order to execute a machine learning algorithm 305running on server 112. This determination and configuration may includeidentification and analysis of: 1. Usage patterns that emerge with thetemplates within specific courses, and the results of such usage, inorder to observe and analyze the student behavior to identify contentelements that will predict better performance for any new course that iscreated; and 2 Dependent and independent variables used to train themachine learning model and algorithm. In some embodiments, presenting ahigher volume of data may create higher potential for identifying theseusage behavior patterns, which may, in turn, provide a more clearevaluation of those content structure elements of the learning coursethat result in higher scores.

Non-limiting examples of such usage behavior pattern data mayinclude: 1. Grades or scores, such as the Check Your Understanding (CYU)grades described herein; 2. Layout or other content structureinformation, such as the lesson presentation template details, or othercontent structure details available within the course authoring tool andspecific content page information created by the content managementsystems disclosed herein; 3. Time spent by students on a course contentpage, or the elements of the course content page, such as a determinedtime spent by a student on an LP page.

Server 112 may further use the collected data from the agents toidentify dependent and independent variables within the collected data.These independent and dependent variables may be used by the server totrain the machine learning model in order to predict grades, such as CYUgrades, when the independent variable information is derived after acontent creator creates a newly created course, as described below.

Non-limiting examples of dependent and independent variables from theanalyzed data may include, as non-limiting examples in some embodiments,dependent variables such as an average CYU grade for a topic, orindependent variables, which may include details of course contentlayout such as text length, image area, time spent on the content orcomponents of the content, details of supplemental links used etc.

Using the aggregated data, including CYU scores, user interaction times,and one or more elements associated with the course content, server 112may convert this aggregated data into a feature set and/or featurevector in preparation for inputting this data into the machine learningmodel 310 in order to execute the machine learning algorithm 305. Theserver may then use this feature set and/or feature vector as parametersto be input into the machine learning model 310.

Server 112 may then input the parameters into the machine learning model310 and execute the machine learning algorithm 305. In some embodiments,in order to create the predictive analytics for course contenttemplates, the machine learning algorithm 305 may apply one or more datascience regression methods to the input parameters derived from thereceived characteristics and features. Specifically, this input may beused to derive the relationship pattern between the independentvariables and the dependent variable. In other words, the machinelearning algorithm may derive the relationship between the grades andscores (the dependent variables), and course content structures, such aslayout or elements, and time information (the independent variables).

Server 112 may then be configured to translate the output of the machinelearning algorithm 305 as results, into scores associated with eachelement of the course content. For example, the output of the machinelearning algorithm 305 may provide scores and/or predictive analytics.These scores may be associated with a set of templates. The disclosedsystem may average and predict CYU grades for a topic that includesseveral LP pages. Server 112 may then store the results of this outputwithin data storage 110.

Once the predictive analytic data is stored within database 110, a user,possibly a learning course instructor, may then access and authenticateto the disclosed system. In response to the instructor user'sauthentication, server 112 may then generate a GUI for accessing anadministration program within the course authoring tool, allowing theinstructor user to access the stored data.

The GUI may be merged into the disclosed system, and the contentcreators who utilize these GUIs may see the data visualizations. Usingthis pattern, the disclosed system may provide the user with means toselect a learning course to create, and/or training data filtrations. Asinstructors select set of parameters from the GUI for a specific class,server 112 may access the stored data resulting from the output of themachine learning algorithm for that learning course and/or itscomponents, and may run the algorithms to generate the output, based onthe received data, user behavior pattern data, and/or dependent andindependent variable data received as input, aggregated, and stored. Themachine learning algorithm 305 may then use existing data for thatcourse as a test data set to generate predictive analytics for theselected course.

The output of the machine learning algorithm 305 may further include oneor more predictions for grades for the newly created course.Specifically, server may utilize the output of the machine learningmodel/algorithm to predict grades, such as CYU grades, based on theindependent variable information derived for a newly created course. Inother words, the disclosed embodiments may predict expected average CYUgrades or other scores for various templates or template elements withinthe selected and newly created learning course.

Using this data, server 112 may update the GUI to display the results ofthe predictive calculations for each of the available content structuresfor the selected and created course. As a non-limiting example, the ThisGUI may include results and a report of the predicted analytics for allavailable templates/layouts or other course structure available to thecourse content creator including details about how the students performagainst the course content created, on average, in an informativemanner. The report further provides a means for the course contentcreator to predict grades (e.g., CYU grades) based on the independentvariable information derived for the newly created course. In someembodiments, the display described above may include a UI widget used toview available recommendations for improving the newly created courseand course content.

The report may also include details such as: time spent on pages, timespent with individual elements, ways that the course or course contentmay be improved, low predictions and the reason that they are low,weekly behavior, how to improve templates (e.g., what elements workbest, which templates, etc.) etc. This report/display including themachine learning results may allow the course content creator identify,at an early stage, the best layout or structure of the content, and/ormedia types to use along with the subject or course domain, as well asany bad content designs, as a preventative measure, thereby avoiding theneed to rectify bad content designs later on. This will result highergrades for learners, and increase the trust on the course materials.

Turning now to FIG. 6, a non-limiting example of this process isdemonstrated. In step 600, the GUI may transmit query parameters (e.g.,the identified characteristics within the course content data) to theprediction engine (e.g., the machine learning algorithm 305 and/or themachine learning model 310). In step 610, the machine learning algorithmmay access data within the database 110 (e.g., the machine learningmodel 310 stored in the database 110, or any relevant course contentdata stored in the database). In step 620, the prediction engine mayprocess all of the received data, or a filtered set of data based on thequery parameters sent from the GUI according to the user preference(e.g., the received course content elements, the converted data tofeature sets/arrays, the course content data from the database or themachine learning model 310), and generate predictions of theeffectiveness of each element within the course content. In step 630,the prediction engine may store the results of this processing andprediction in the database 110. In step 640, a prediction dashboard(e.g., the GUI referenced above) may make a request for the results datafrom the prediction engine. In step 650, the prediction data may beselected from the database and transmitted to the client device fordisplay on the prediction dashboard.

Turning now to FIGS. 7-9, different displayed data from this process isdemonstrated. For example, FIG. 7 demonstrates a non-limiting exampledisplay of the CYU data resulting from the process steps above. FIG. 8demonstrates a non-limiting example display of the content time usagedata resulting from the process steps above. FIG. 9 demonstrates anon-limiting example display of the prediction data resulting from theprocess steps above.

Non-limiting examples of use cases for this scenario are as follows: 1.A course content creator may want to know the predicted grades for thecourse he/she created. The disclosed embodiments may predict the CYUscores; 2. A course content creator may see the recommendations oncourse content structure/layout in order to get higher grades.

Other embodiments and uses of the above inventions will be apparent tothose having ordinary skill in the art upon consideration of thespecification and practice of the invention disclosed herein. Thespecification and examples given should be considered exemplary only,and it is contemplated that the appended claims will cover any othersuch embodiments or modifications as fall within the true scope of theinvention.

The Abstract accompanying this specification is provided to enable theUnited States Patent and Trademark Office and the public generally todetermine quickly from a cursory inspection the nature and gist of thetechnical disclosure and in no way intended for defining, determining,or limiting the present invention or any of its embodiments.

The invention claimed is:
 1. A system comprising: a database coupled toa network and storing: a course content; and a machine learning modelconfigured to generate a recommendation for an update to the coursecontent; a first client device coupled to the network and running acourse content activity agent comprising at least one software moduleconfigured to identify, in association with the course content, acontent usage data comprising: an assessment score; and a userinteraction time data; a server comprising at least one computing devicecoupled to the network and comprising at least one processor executinginstructions within memory which, when executed, cause the system to:receive, though the network from the course content activity agent, thecontent usage data; translate the content usage data into at least onecourse content activity parameter; input the at least one course contentactivity parameter into the machine learning model; generate, using anoutput from the machine learning model, a graphical user interface(GUI), transmitted through the network and displayed on a second clientdevice, comprising: a report of the output; a recommendation for theupdate to the course content; and a prediction, in response to theupdate, of an increase to an average assessment score associated withthe course content.
 2. The system of claim 1, further comprising aJavaScript file, executed in association with an HTML page comprisingthe course content, the JavaScript file including the course contentactivity agent.
 3. The system of claim 1, wherein the instructions, whenexecuted, further cause the system to identify, in association with thecourse content a plurality of course content element data comprising atext length, an image area, an external supplemented added factor, anexternal image or number of external images, an external link or numberof external links, an external video or number of external videos, anexternal Uniform Resource Locator (URL) or number of URLs, or anexternal textual document or number of external textual documents. 4.The system of claim 1, wherein the instructions, when executed, furthercause the system to associate each element of the course content with anidentification selected from the group consisting of a courseidentification, a page identification, a lesson identification, atemplate identification, or course content element identification. 5.The system of claim 1, wherein the instructions, when executed, furthercause the system to determine the user interaction time data accordingto a load time of a lesson presentation page and an unload time of thelesson presentation page.
 6. The system of claim 1, wherein theinstructions, when executed, further cause the system to analyze andmake determinations from the content usage data received according to:at least one content usage pattern within the content usage data; and atleast one dependent variable identified within the content usage datacompared to at least one independent variable identified within thecontent usage data.
 7. A method comprising: storing, by a servercomprising at least one computing device coupled to a network andcomprising at least one processor executing instructions within memory,within a database coupled to the network: a course content; and amachine learning model configured to generate a recommendation for anupdate to the course content; receiving, by the server, though thenetwork from a course content activity agent running on a first clientdevice, a content usage data, identified in association with the coursecontent, and comprising: an assessment score; and a user interactiontime data; translating, by the server, the content usage data into atleast one course content activity parameter; inputting, by the server,the at least one course content activity parameter into the machinelearning model; generating, by the server using an output from themachine learning model, a graphical user interface (GUI), transmittedthrough the network and displayed on a second client device, comprising:a report of the output; a recommendation for the update to the coursecontent; and a prediction, in response to the update, of an increase toan average assessment score associated with the course content.
 8. Themethod of claim 7, further comprising the step of executing a JavaScriptfile in association with an HTML page comprising the course content, theJavaScript file including the course content activity agent.
 9. Themethod of claim 7 further comprising the step of identifying, by theserver in association with the course content, a plurality of coursecontent element data comprising a text length, an image area, anexternal supplemented added factor, an external image or number ofexternal images, an external link or number of external links, anexternal video or number of external videos, an external UniformResource Locator (URL) or number of URLs, or an external textualdocument or number of external textual documents.
 10. The method ofclaim 7, further comprising the step of associating, by the server, eachelement of the course content with an identification selected from thegroup consisting of a course identification, a page identification, alesson identification, a template identification, or course contentelement identification.
 11. The method of claim 7, further comprisingthe step of determining, by the server, the user interaction time dataaccording to a load time of a lesson presentation page and an unloadtime of the lesson presentation page.
 12. The method of claim 7, furthercomprising the steps of analyzing and making determinations, by theserver, from the content usage data received according to: at least onecontent usage pattern within the content usage data; and at least onedependent variable identified within the content usage data compared toat least one independent variable identified within the content usagedata.
 13. A system comprising a server, comprising at least onecomputing device coupled to a network and comprising at least oneprocessor executing instructions within memory, the server beingconfigured to: store, within a database coupled to the network: a coursecontent; and a machine learning model configured to generate arecommendation for an update to the course content; receive, though thenetwork from a course content activity agent running on a first clientdevice, a content usage data, identified in association with the coursecontent, and comprising: an assessment score; and a user interactiontime data; translate the content usage data into at least one coursecontent activity parameter; input the at least one course contentactivity parameter into the machine learning model; generate, using anoutput from the machine learning model, a graphical user interface(GUI), transmitted through the network and displayed on a second clientdevice, comprising: a report of the output; a recommendation for theupdate to the course content; and a prediction, in response to theupdate, of an increase to an average assessment score associated withthe course content.
 14. The system of claim 13, wherein the server isfurther configured to execute a JavaScript file in association with anHTML page comprising the course content, the JavaScript file includingthe course content activity agent.
 15. The system of claim 13, whereinthe server is further configured to identify, in association with thecourse content, a plurality of course content element data comprising atext length, an image area, an external supplemented added factor, anexternal image or number of external images, an external link or numberof external links, an external video or number of external videos, anexternal Uniform Resource Locator (URL) or number of URLs, or anexternal textual document or number of external textual documents. 16.The system of claim 13, wherein the server is further configured toassociate each element of the course content with an identificationselected from the group consisting of a course identification, a pageidentification, a lesson identification, a template identification, orcourse content element identification.
 17. The system of claim 13,wherein the server is further configured to determine the userinteraction time data according to a load time of a lesson presentationpage and an unload time of the lesson presentation page.
 18. The systemof claim 13, wherein the server is further configured to analyze andmake determinations from the content usage data received according to:at least one content usage pattern within the content usage data; and atleast one dependent variable identified within the content usage datacompared to at least one independent variable identified within thecontent usage data.